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Abstract—Metacognition is a valuable tool for learning, since 
it is closely related to self-regulation and awareness of one’s 
own affect. However, methods for automatically detecting and 
studying metacognition are scarce. Thus, in this paper we 
describe an algorithm for automatic detection of metacognitive 
language in writing. We analyzed text from the forums of two 
online, university-level science courses, which revealed common 
patterns of phrases that we used for automatic metacognition 
detection. The algorithm we developed exhibited high accuracy 
on expert-labeled metacognitive phrases (Spearman’s rho = 0.878 
and Cohen’s kappa = 0.792), and provides a reliable, fast method 
for automatically annotating text corpora that are too large 
for manual annotation. We applied this algorithm to analyze 
relationships between students’ metacognitive language and their 
academic performance, finding small correlations with course 
grade and medium-sized differences in metacognition across 
courses. We discuss how our algorithm can be used to advance 
metacognitive studies and online educational systems. 

Index Terms—Metacognition, learning analytics, natural lan- 
guage processing 


I. INTRODUCTION 


Most of us have thought about our own thinking. We 
often ask ourselves if an idea is worth pondering over, what 
to think about next, how we arrived at a flawed conclu- 
sion, or if we truly understand a certain idea. While such 
metacognition, or “thinking about thinking”, is typically only 
a tool intended to assist the thinker, humans often behave in 
ways that communicate signals of their metacognition. In a 
student-teacher interaction context, for example, teachers may 
assess metacognition through affective and behavioral cues 
[1]. Evaluating such expressions of student thinking is also 
a useful tool for the teacher in this case because they can 
assess what the student may or may not understand, and adapt 
their methods of teaching accordingly [2], [3]. In this paper, 
we present a method for automatic detection of metacognitive 
language in computerized discussion forums. We apply this 
method to online STEM (science, technology, engineering, and 
math) courses, and discuss how it will enable further large- 
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scale research on the interplay between metacognition, affect, 
and learning. 

Metacognition is awareness of one’s own thinking [4]. 
More specifically, it is the process used to plan, monitor 
and assess one’s understanding and thought process [5]. 
Hence, metacognition is especially important for self-regulated 
human-computer interaction tasks such as learning with tech- 
nology [2], [6]. While metacognition is primarily internally- 
accessible, people may externally indicate their metacognition 
through bodily gestures (e.g., putting a finger to the chin to 
indicate thought), conversational cues (e.g., “hmmmmm’, “let 
me think a second”) [7], or written language (e.g., descriptions 
of the thought process in narrative answers to homework 
exercises) [8]. In educational contexts, teachers can use these 
indicators of metacognition to understand what students need 
to work on, which allows teachers to adapt and improve their 
pedagogy. 

Methods for automatic detection are key for enabling large- 
scale analysis of metacognition in learning contexts (and other 
discussion-based activities outside the scope of this paper). 
Detecting metacognition, however, is a nontrivial task. In 
order to detect metacognition, one must be able to infer some 
meaning from context. In a parsing algorithm, one of the major 
difficulties stems from words having different meanings de- 
pending on its usage and context. Common research methods 
employ time-intensive methods in which either the researcher 
conducts post-activity inquiries to analyze metacognition [9], 
[10], or participants are required to self-report moments of 
metacognitive awareness [11]-[15]. Thus, in this paper we 
focus on automatic detection of metacognitive phrases in bod- 
ies of student-generated text in ecological contexts (students’ 
daily learning activities). Specifically, we analyze students’ 
metacognition in online STEM courses, though our methods 
are not limited to these contexts. 

Our research is novel in that we provide an algorithm for the 
automatic detection of metacognition in bodies of text obtained 
from online discussion forums. By allowing researchers to 
gather large amount of metacognitive data quickly and ac- 
curately, our algorithm can reliably accelerate metacognitive 
research endeavors. We also discuss later how students and 
teachers can use automatic detection of metacognition to 


improve learning. 


II. RELATED WORK 


We focus the discussion of related work illustrating the 
importance of studying metacognition, and examples of related 
studies that have explored automatic detection of cognitive and 
affective states related to metacognition. 


A. Metacognition, cognition, and affect 


Simply put, metacognition is thinking about thinking. It is 
understanding how much and what information is needed to 
reach a goal, being conscious of a strategy for obtaining the 
information required to reach the goal, and thinking about 
when and how to use the available information to achieve that 
goal [5]. Although easily confused, cognitive strategies and 
metacognitive strategies are distinct in an educational context. 
For example, cognitive strategies include — but are not limited 
to — rehearsal, elaboration, organization, and critical thinking, 
while metacognitive strategies deal with time management, 
self-evaluation, concentration/effort, and self-awareness [16]. 
While cognitive strategies are critical for making progress and 
building knowledge, metacognitive strategies oversee these 
processes and permit improvement to processing (via moni- 
toring), making metacognitive knowledge critical for cognitive 
strategies to be performed well [4]. 

A large body of research exists that shows differences 
between people who possess and/or exercise metacognition 
and those who do not. Pennycook, Ross, Koehler, & Fugelsang 
[15] suggested that people who engaged in using metacog- 
nition were usually analytic thinkers rather than intuitive 
thinkers, a framework that is essential to understanding STEM 
topics. It is metacognitive thinking and strategies that have 
been shown to improve learning outcomes [17], [18] across the 
board [19], and have been shown to be the differing variable 
between high-achievers and low-achievers [6], [20]—[22]. 

Previous work has also noted that there are key relation- 
ships between metacognition and affect. First, self-reports of 
emotion (which are common in human-computer interaction 
studies [23]) are correlated with metacognition [24]. Second, 
metacognitive self-awareness is important for affective self- 
regulation as well as cognitive self-regulation [25], [26]. These 
connections between metacognition and affect explain some of 
the relationships between affect and learning [25], [27], [28], 
and highlight the importance of further understanding the role 
of metacognition in current online learning environments. 


B. Metacognition and learning 


A large body of previous research on metacognition in- 
dicates that successful self-regulated learning is conditioned 
on students’ participation in metacognitive activities [2], [11], 
[29]-[32]. As an educational intervention, the implementation 
of metacognition has been very successful. Hadie et al. [3] 
demonstrated that when a traditional lecture was reconstructed 
so that it contained metacognitive thinking cues, students 
significantly improved their learning. Casey, Gill, Pennington, 
& Mireles [33] showed under-performing low-income English 


Language Learners how to successfully program robots, as- 
sisted by metacognitive strategies. Metacognitive elements, 
like knowing that one does not know something and why they 
do not know it, are critical in fixing mistakes and dispelling 
faulty conclusions, especially in educational contexts. Further- 
more, metacognition has also been shown to be associated with 
willingness to adjust faulty learning approaches [34]. 


C. Automatic detection of metacognition 


As previously mentioned, most research on metacognition 
in the educational context consists of self-reports [10], [14], 
interventions [8], [35], and observations [1], [36], [37]. While 
extremely informative, these methods prevent researchers from 
analyzing spontaneously-produced metacognitive processes on 
a large scale and in a natural, generalizable setting with no 
disruptions. To our knowledge, there is no existing research 
on automatic detection of metacognitive language nor is there 
research that concerns large-scale analysis of metacognitive 
language. 

Many natural language processing (NLP) methods have 
been employed to detect related cognitive and affective con- 
structs in student-produced free-form text, however. As pre- 
viously mentioned, automatic detection from natural language 
input has potential to be a more accurate, cost-effective, and 
less time-consuming alternative to self-reports and researcher- 
annotated responses. Given these advantages, NLP-based auto- 
matic detection of cognition and affect is in mainstream usage, 
including in automatic essay graders like AES Systems [38], 
Project Essay Grader [39], Latent Semantic Analysis [40], 
and E-rater [41]. Similarly, NLP-based affect detection has 
shown promise for evaluating learning [42]. Such methods of 
automatic detection have proven to be extremely beneficial in 
the educational context. 

The possibility of applying NLP methods to analyze char- 
acteristics of learning and learners is also feasible, given 
their accuracy and relationships to learning. Lintean, Rus, & 
Azevedo [13] investigated NLP methods on student-articulated 
paragraphs with hopes of being able to use the paragraphs 
to describe and better understand students’ mental models. 
They presented several approaches for automatically detecting 
language to describe students’ mental models in an online 
tutoring environment called MetaTutor. Their study took place 
during a self-regulatory activity of prior knowledge activation 
in which the students had to type a paragraph on their prior 
knowledge of the topic at hand. The results revealed that 
a word-weighting method using TF-IDF (term frequency— 
inverse document frequency) features calculated from the 
text corpus, combined with a Bayes Net machine learning 
algorithm, yielded the most accurate results. 

While researchers have explored automatic detection of 
affect and cognition in various learning domains including 
Intelligent Tutoring Systems (ITSs), massive open online 
courses (MOOCs), and others. Lintean et al.’s work [13] is 
some of the only work to date (of which we are aware) 
that automatically analyzes student-produced text dealing with 
metacognition — though their object was to describe and 


understand cognition, rather than metacognition, in an edu- 
cational context designed to promote metacognition. Previous 
research has also studied automatic detection of constructs 
related to or including metacognition, such as self-caught 
mind wandering [12], [43]—[45]. In contrast, our research is 
novel in that we provide an automatic detection method for 
metacognitive phrases. Specifically, we capture phrases that 
indicate confident and unconfident metacognition, to enable 
future research into the role of metacognition during text-based 
interactions with computers. 


III. METHOD 
A. Discussion forum post data 


Data we analyzed to develop and test the metacognition 
detection method came from two introductory STEM courses 
— one with a single semester of data (course 1), and another 
with several semesters of data (course 2) — at a large, pub- 
lic university in the Midwestern United States. These data 
included all of the students’ discussion forum posts as well 
as their final course grades, which were provided to us by 
university data curators on an coarse-grained ordinal scale to 
preserve student privacy. Specifically, there were four levels of 
grades: A- to A+, B- to B+, C- to C+, and D+ or lower. For 
correlation analyses involving grades, we represented the four 
grade levels as integers 0 through 3, where higher numbers 
indicated better grades. 

In both courses, forum participation was required as part 
of students’ participation grades. Students were required to 
regularly post questions they had, or to answer other students’ 
questions. This requirement may have resulted in increased 
participation (and different quality of participation) relative 
to online courses with optional forum participation. However, 
some students did not participate regularly despite this require- 
ment. 


B. Participants 


We received university ethics board approval before ob- 
taining data analyzed in this paper. Data were retrospectively 
analyzed (with permission from instructors), to avoid poten- 
tially influencing instructor perceptions or students’ grades 
for ongoing courses. University data curators preprocessed all 
discussion forum posts by replacing student names, places, 
and other identifying information with placeholders in order 
to protect their identities. In total, we obtained 19,700 forum 
posts for analysis, from 710 students who made at least one 
discussion forum post. 


C. Preliminary analysis of metacognition 


We began by analyzing a random sample of 200 forum posts 
to find patterns of metacognitive phrases. We found that most 
metacognitive phrases began with a pronoun and ended with 
a thought-related word (e.g., realized, considered, thought). 
We were interested in detecting metacognitive statements 
that described students’ own metacognitive processes, so we 
limited pronouns to first person only (e.g., J, we, our). In 
this initial sample we also noted that metacognitive phrases 


largely fell into two categories: expressions of a student’s 
thoughts regarding an idea (which we refer to as confident) and 
expressions of doubt about their thoughts (which we refer to as 
unconfident). With few exceptions, metacognitive phrases be- 
gan with a first-person pronoun and ended with a confident or 
unconfident thought-related word, and occasionally included 
negating words in between (e.g, no, not). We defined “normal” 
metacognitive phrases as those which followed this pattern. 

Our metacognition detection method primarily focuses on 
capturing this normal pattern of metacognitive phrases, but 
also captures a select set of common exceptional metacognitive 
phrases that do not follow the normal pattern. 


D. Automatic metacognition detection algorithm 


We categorized words into 4 main classes, with some classes 
containing their own respective subclasses: 
1) Pronouns 
e First person: (e.g., i, we, my) 
e Non-first person: (e.g., you, him, she) 
2) Negations: (e.g., not, no) 
3) Metacognitive 
e Confident: (e.g., understand, know, believe) 
e Unconfident: (e.g., confused, unsure, struggle) 
4) Other: all other words that do not fall into the first three 
classes 
We created a dictionary set of words for each class/subclass 
to determine whether or not a given word belongs in that 
category (words that did not appear in any dictionary set we 
classified as Other). 
Let w be the list of words for a given body of text (e.g. w = 
[i,am, sure, you, are, right]). We defined the set of normal 
metacognitive phrases within w as 


{wl i+ |] | wii] © pronouns first, 


wii + j] © metacognitive, 
wlk] ¢ pronouns.nonfirst Vk,i< k <i +9, 


vijiisng ssh 


For example, if w = [i, am, sure, you, are, right], then the 
set of normal metacognitive phrases would be 


NMP = {w/[1 : 3] = [i, am, sure]} 


We ensured that phrases within this set did not overlap with 
each other in the original body of text by left association 
(we searched for phrases from left to right). We also limited 
the maximum length of normal metacognitive phrases to five 
words, as we found this was sufficient to capture virtually all 
metacognitive phrases of this pattern (see Fig. 1). 

Normal metacognitive phrases were further classified into 
confident and unconfident based on whether the number of 
negations in the phrase is odd or even, as well as the subclass 
of the ending metacognitive word. For example, a normal 
metacognitive phrase with one negation ending with an un- 
confident metacognitive word would be classified as confident 
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Fig. 1. A histogram of metacognitive phrase lengths (in terms of number of 
words) as captured by our detection algorithm 


and a phrase with three negations that ends with a confident 
metacognitive word would be classified as unconfident. 


In order to fully capture all metacognitive phrases, we 
also created two whitelists of regular expressions, one for 
confident and another for unconfident, that did not follow the 
normal pattern — such as acronyms or adages. One example is 
IMO, an abbreviation for “in my opinion”, which can indicate 
metacognitive awareness. 


The largest difficulty in developing the algorithm was cu- 
rating the dictionary word lists that altered the behavior of the 
algorithm. Words like “lost” were often too ambiguous to be 
considered metacognitive, and so we chose to capture common 
phrases containing those words with the regular expression 
whitelists. In total, we analyzed 1000 posts to curate the 
dictionary word lists and assess the algorithm. Fig. 2 and 
Fig. 3 illustrate the results of applying the algorithm to two 
randomly-chosen forum posts. 


[REDACTED], Although your estimate doesn’t 


a very valid point. 
that the star formation rate was much faster 
long ago than it is now. Many of the stars in 
the galaxy that we can see today could very 
well have been born billions of years ago. 
It’s a smart idea to consider not just rely on 
in th 
in the galaxy that occur 


what w 


can s sky, but on the changes 


Although with this 


new information I ott what my new 
but |I believe that there 


is undoubtedly merit in your reasoning 


estimate would be, 


Fig. 2. Algorithm’s annotation of metacognitive language from one example 
forum post, with confident phrases highlighted in green and unconfident 
phrases in red. 


Excellent job [REDACTED]! As we have had to do 


in the past for 


stimates, we tend to rely on 


what we know from Earth for it is our only 


known example. Seeing as we have just started 


to scratch the surface of communication 


I believe we will be communicating for 


many more years. My estimat 


may b 
than yours but |we both had the same thought 
process deciding our estimate of L! 


larger 


Fig. 3. Another example of automatic annotation on a forum post with only 
confident phrases (highlighted in green). 


E. Algorithm assessment 


As stated before, we began by randomly sampling 200 
forum posts. We had two researchers who were familiar 
with metacognition research literature — but not with the 
algorithm — evaluate and label the true number of confident 
and unconfident phrases for 100 of these posts (half of the 
sample). The annotators resolved their disagreements, and we 
utilized their annotations to develop the first iteration of the 
automatic labeling algorithm. The annotators then labeled the 
second half of the sample, which we used as the initial test of 
the prototype algorithm. 

Within this second sub-sample of 100 posts, the algorithm 
achieved a Cohen’s « of 0.770 with a 95% confidence in- 
terval (CI) of [0.684,0.855], and Spearman’s p correlation 
of 0.871 (CI = [0.815,0.912]) for positive metacognitive 
phrases [46], [47]. The algorithm achieved Cohen’s « = 0.804 
(CI = [0.665,0.943]) and Spearman’s p = 0.817 (CI = 
(0.740, 0.873]) for negative metacognitive phrases. 

However, the confidence intervals were relatively large 
(exceeding +0.100 for kappa, for example). By scaling our 
sample size through repetition (Fig. 4 and Fig. 5), we extrap- 
olated that another test of 400 posts were needed to assess 
the algorithm’s quality with a sufficiently tighter confidence 
interval (well under +0.100 in all cases). Thus, the annotators 
labeled another 400 randomly-chosen posts, and we made 
algorithmic refinements based on the second half of the 200- 
post sample. 

From this second round of testing with 400 posts, we indeed 
achieved a much tighter confidence interval with « and p 
values that indicate large effect sizes (at least 0.800 in all 
respects) [48]. 

During the second round of testing, we found it necessary 
to modify the lists of words and phrases — predominately for 
unconfident metacognitive phrases. Given that we modified the 
algorithm, we completed another round of manual annotation 
and testing following the same procedure, again with 400 
forum posts. This last sample of 400 posts represents the test 
set, which we analyzed for our final results. 


IV. RESULTS 


The focus of this paper is on automatic detection of 
metacognition. However, in these results we also consider 
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Fig. 4. An extrapolation of the « confidence interval from an initial test on 
100 sample discussion forum posts. 
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Fig. 5. An extrapolation of the p confidence interval from an initial test on 
100 sample discussion forum posts. 


example applications of the metacognition detection method 
to demonstrate its utility as a measure. 


A. Metacognition detection accuracy 


Our metacognitive language detection algorithm demon- 
strated high accuracy in predicting the number of confident 
and unconfident metacognitive phrases in bodies of text. We 
compared the finalized algorithm to two human annotators on 
a held-out test set of 400 forum posts from an online STEM 
course (course 1). Each human rater independently annotated 
the posts and resolved any disagreements via discussion with 
a third rater to create a consensus on the ground truth anno- 
tations. Finally, we calculated the agreement between the two 
human raters (before disagreement resolution) as well as the 
agreement between the algorithm’s output and the resolved 
ground truth annotations. 

We found that our algorithm’s accuracy versus the ground 
truth annotations was comparable to the agreement between 
the two human raters (Table I), who were themselves highly 
reliable [48]. Finally, when applied to the entire dataset from 


both courses (19,700 forum posts), the algorithm detected 
23,466 metacognitive phrases (20,092 confident and 3374 
unconfident). 


TABLE I 
SPEARMAN’S RHO AND COHEN’S KAPPA AGREEMENT BETWEEN OUR 
ALGORITHM AND HUMAN ANNOTATORS (AFTER CONSENSUS), AS WELL 
AS BETWEEN HUMAN RATERS PRE-CONSENSUS ON THE NUMBER OF 
METACOGNITIVE PHRASES PER FORUM POST (ALL p < .001). 


Algorithm accuracy Human agreement 


Metacognition type p K p K 

Confident 861 .776 893 810 
Unconfident 859 825 .786 72 
Either 878 .792 899 822 


B. Correlations with learning outcomes 


We applied the detection algorithm to all data from both 
courses and examined correlations between metacognitive 
language usage and final course grade (a measure of success). 
In particular, we considered the mean number of metacognitive 
phrases per forum post for each student, as well as means 
for confident and unconfident phrases separately. Grade levels 
were ordinal variables, so we computed rank-order correlations 
(Spearman’s ) for these comparisons. 

Results in Table II show that there were small correlations 
between metacognition and grade within each of these courses. 
In course 1, confident metacognitive phrases and overall 
metacognitive phrase usage were positively correlated with 
grade, indicating higher grades for students who used more 
metacognitive language. In course 2, unconfident metacogni- 
tive phrases were positively related to grade, though each of 
these effect sizes was small. 

Conversely, effect sizes for the relationship between 
metacognition and grade were medium-sized [48]. In the next 
section, we explore this notable difference in more detail. 


TABLE II 
SPEARMAN CORRELATIONS BETWEEN METACOGNITIVE PHRASE 
OCCURRENCES (PER POST) AND FINAL GRADES. * INDICATES 
SIGNIFICANT CORRELATIONS (p < .05). 


Course Confident » Unconfident p Either p 
Course 1 .158* 128 .166* 
Course 2 .077 .090* 071 
Combined 473* 358* .468* 


C. Metacognitive differences across courses 


Correlations between metacognition and grade were surpris- 
ingly large when combining data from both courses (9 = 
.398, .473, and .468 for confident, unconfident, and total 
metacognition, respectively). On the other hand, within-course 
correlations (i.e., controlling for course) were comparatively 
low (Table II). We investigated this pattern further and found 


that the contrasting correlations were attributable to differences 
between the two courses. 

As shown in Table III, students in course | employed sig- 
nificantly more metacognitive phrases than students in course 
2. Furthermore, the course | students received significantly 
higher grades than course 2 students, thereby explaining 
the substantial correlations between metacognition and grade 
when analyzing both courses combined!. 


TABLE III 
METACOGNITION AND GRADE DIFFERENCES ACROSS COURSES. GRADES 
ARE ON A [0,3] SCALE, WHERE HIGHER NUMBERS ON THE SCALE 
INDICATE BETTER GRADES. ALL DIFFERENCES BETWEEN COURSES WERE 
STATISTICALLY SIGNIFICANT (p < .001). 


Measure Course 1 mean Course 2 mean 


Metacognitive phrases per post 


Confident 1.393 0.205 
Unconfident 0.203 0.092 
Either 1.596 0.297 
Grade 2.662 1.381 


V. DISCUSSION 
A. Main findings 


We were interested in the problem of automatic detection 
and annotation of metacognitive language in computerized 
educational discussion forums. To approach this problem, we 
examined human-annotated forum text and developed a nat- 
ural language processing approach that locates metacognitive 
phrases in bodies of text with high accuracy. 

We also measured correlations between metacognitive lan- 
guage and academic performance. While we expected a 
positive relationship, we found relatively small correlations 
between metacognitive language and academic performance 
within each course. On the other hand, when the courses were 
combined, correlations between metacognition and grades 
were much larger, which suggests important differences be- 
tween courses. Possible explanations for this could include 
the reasons students had for taking a particular course, course 
structure/curriculum, student demographic differences, influ- 
ence of forum participation on grade, etc. These course dif- 
ferences will be the subject of our further research. 

Our algorithm can also be used in future metacognitive 
research to automate text analysis of metacognitive language. 
When there is a relatively large amount of data, humans 
may incur error from fatigue, inexperience, or other causes — 
whereas our algorithm is inherently consistent, efficient, and 
validated on a large sample of reliable expert annotations. 


‘Differences in metacognition between courses remain similar after con- 
trolling for length of forum posts; thus, correlations between metacognition 
and grade remain as well. 


B. Limitations and future work 


While our metacognition detection method exhibited high 
agreement with human raters, its ability to generalize to 
notably different sources of text remains unclear. For example, 
individuals who are much older or younger than typical uni- 
versity students may express metacognition differently. Future 
work is needed to measure generalization at different levels 
including across universities, ages, demographic characteris- 
tics, and other dimensions. Additionally, our data exclusively 
consisted of English-language text, and our current detection 
model would certainly not generalize to other languages. 

Our algorithm looks for metacognitive language by finding 
specific patterns of particular words. It is thus primarily limited 
by the words in the word lists. Therefore, in future work 
we plan to address issues of generalization, as they arise, 
by modifying these lists that help the algorithm categorize 
metacognitive phrases. 

Another alternative solution is deep learning, a popular 
method with many successful applications. While our solution 
already achieves high correlations, it might be worth training 
a neural network to compare against our solution. 

In future work we also intend to apply the algorithm 
to a variety of online STEM courses, with the goal of 
identifying metacognitive differences between students and 
courses that might lead to educational improvements. For 
example, interventions intended to improve learning outcomes 
via metacognitive interventions might likely be much more 
valuable for students who are not already evaluating their own 
knowledge and thought processes. 


C. Concluding remarks 


Metacognitive research is limited by the amount of data 
we can gather. Current research relies on human annotation, 
which can be time consuming and inconsistent across research 
endeavors. Our algorithm consistently detects metacognition 
from text generated during computer-mediated discussion, and 
has shown high accuracy in a large, held-out test set (400 
posts). This method will enable future research to quickly 
measure metacognitive language usage, thereby enhancing 
our understanding of the relationships between metacognition, 
affect, and learning. Toward this goal, we have made the 
source code of our algorithm publicly available”, as well as a 
web interface to enable users to easily apply the algorithm to 
their own data files. 
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